A heads-up display is generally defined as an electronically generated display containing information or data that is superimposed on an observer's normal field of view. As explained in greater detail below, heads-up display (“HUD”) systems have been used in various applications. One such application is for use by pilots of aircraft. In the typical aircraft HUD system, a semi-transparent display screen is located generally in front of the eyes of the pilot (i.e. a screen mounted on the pilot's head or helmet, or in the view of the aircraft windshield). Such a system enables a pilot to concentrate on the difficult tasks associated with flying the aircraft, without diverting his attention to scan or examine a wide array of instruments.
It is also well known that medical technicians or personnel, such as surgeons, must keep track of many different types of information during an operation. For example, a surgeon must carefully view or monitor the physical surgery while simultaneously monitoring a patient's condition (e.g., blood pressure, heart rate, pulse, etc.). In addition, depending on the procedure, the surgeon must also monitor the status and settings of surgical equipment and tools. Although the additional information is necessary and important, monitoring such information often diverts the surgeon from the immediate task at hand.
Some surgical operations require that the surgeon divert his eyes to view a video monitor, for example, when performing highly complex laser or internal surgery conducted through scopes. See U.S. Pat. No. 5,222,477, which discloses an endoscope or borescope stereo viewing system. In addition, the surgeon may from time to time need to refer to data, such as defined by a patient's recorded or written history, or to previously taken x-rays or other computer generated images (e.g., CAT, NMR, 3D, etc.). For example, U.S. Pat. No. 5,452,416 discloses an automated system and a method for organizing, presenting, and manipulating medical images for viewing by physicians. See also U.S. Pat. Nos. 5,251,127 and 5,305,203, which disclose a computer-aided surgery apparatus that positions surgical tools during surgery or examinations. In each of the above-described systems, in order to view the displayed information, the surgeon must divert his or her eyes to a remote monitor.
Thus, surgeons use many different types of displays and must continually monitor many different sources of information. However, as more tools and sources of data become available to surgeons for use during operations, more opportunities for distraction arise. It is difficult for a surgeon to focus on his or her conduct during a surgical procedure while also continually shifting focus away from the patient to other monitors or indicators. Therefore, a need exists for conveniently, efficiently and accurately displaying to a surgeon various types and sources of information, views, and images of a patient undergoing a critical medical procedure. As explained in greater detail below, prior attempts in the medical field to fulfill that need have been unsatisfactory.
For example, video signal sources have been adapted to scan views or images for different types of medical uses and applications. U.S. Pat. No. 4,737,972 to Schoolman (“Schoolman I”) discloses a head-mounted device that provides stereoscopic x-ray images. Furthermore, U.S. Pat. No. 4,651,201 to Schoolman (“Schoolman II”) discloses an endoscope that provides stereoscopic images of the patient on a display. Both Schoolman I and Schoolman II allows for the selective transmission of other video data to the display. However, Schoolman I and Schoolman II do not use a “see through” display that allows the surgeon to monitor both the environment around him and the video image. If the surgeon wishes to monitor or view the real-world environment, as opposed to the displayed graphics, the head-mounted display must be removed.
Efforts have also been made to use head-mounted displays in augmented reality simulations for medical applications wherein a desired image or three-dimensional model is superimposed on the real scene of a patient. For example, it was reported that a research effort in the Department of Computer Science at the University of North Carolina has attempted to develop a see-through head-mounted display that superimposed a computer-generated three-dimensional image of the internal features of a subject over the real-life view of the subject. Information describing those research efforts may be found on the World Wide Web in a document maintained by Jannick Rolland at the site on the World Wide Web Pages of the NSF/ARPA Science and Technology Center for Computer Graphics and Scientific Visualization at the University of North Carolina, Chapel Hill (http://www.cs.unc.edu/˜rolland, cited February, 1996, copies of which are included in the information disclosure statement that has been filed concurrently with this application). That World Wide Web site in turn referenced the following publication: A. R. Kancheral, et al., “A Novel Virtual Reality Tool for Teaching 3D Anatomy,” Proc. CVR Med '95 (1995). Other research efforts at the University of North Carolina attempted to use a video see-through head-mounted display and a high-performance computer graphics engine to superimpose ultrasound images over the real view of the subject, thereby allowing a user to “see within” the subject. A set of trackers captured the motion of the body part with respect to the field of view of the user, and a computer updated the position of the body part in real time. The computer attempted to correlate the “tracked” position of the body with the three-dimensional model and to display the model on the heads-up display in a manner that gave the appearance of “x-ray vision.”
In the above-described University of North Carolina research efforts, the focus was primarily to help teach students by superimposing a single computer-generated image over a moving, real-life, image of a subject. However, as explained in the associated literature, the “tracking” requirements made the research effort quite complicated, and the results appeared less than satisfactory. Moreover, such a teaching system is not applicable to the real-world environment of a surgeon, where the patient is not moved (and “tracking” is unnecessary), and where the surgeon needs or desires other information to be made readily available for viewing.
Still another research program associated with the University of North Carolina is described in Fuchs, et al., “Virtual Space Teleconferencing using a Sea of Cameras,” Proceedings of the First International Symposium on Medical Robotics and Computer Assisted Surgery (Pittsburgh, Pa., Sep. 22-24, 1994). That article describes research efforts that attempted to use a multitude of stationary cameras to acquire both photometric and depth data. The acquired data was purportedly used to construct a remote site in accordance with the head position and orientation of a local participant. According to the article, each participant wears a head-mounted display to look around a remote environment having surface geometries that are continuously sensed by a multitude of video cameras mounted along the walls and ceiling, from which cameras depth maps are extracted through cross-correlation stereo techniques. Views acquired from several cameras are then displayed on a head-mounted display with an integrated tracking system to provide images of the remote environment. The explained purpose of the effort was to duplicate, at a remote location, a three-dimensional virtual reality environment of a medical room. However, the article does not disclose the use of see-through displays providing a surgeon with the ability to select and display additional forms of data, or to superimpose data over a real-life view of the patient or surgical site.
Another type of head-mounted display is described in Yoshida, et al., “Optical Design and Analysis of a Head-Mounted Display with a High-Resolution Insert,” Proc. SPIE 2537 (1995). That article describes yet another research program associated with the University of North Carolina in which a small area of a high-resolution image is inserted on a large field of a low resolution image displayed on a head-mounted screen. The system is described as using eye-tracking information to dynamically place the high resolution insert at the user's gaze point. The system purports to provide the user with both high-resolution imagery and a large field of view. In essence, using eye-tracking electronics, the “inserted image” corresponding to the user's gaze point is converted from low resolution to high-resolution. However, as above, the user can not select additional or alternative forms of data or different images to be superimposed over the primary image on the head-mounted display.
Thus, few head-mounted displays have been developed for the medical industry, and all those described above have had limited purpose and utility. On the other hand, and as discussed briefly above, a wide variety of head-mounted devices are commonly used in military applications. As mentioned, aircraft pilots, tank commanders, weapon operators and foot soldiers have all used head-mounted displays to display various forms of weapon or image information along with other data defining the real-world environment of the person wearing the display. For examples of such systems, see the following U.S. Pat. Nos. 4,028,725; 5,281,960; 5,000,544; 5,227,769; 4,994,794; 5,341,242; 4,878,046; 3,940,204; 3,923,370; 4,884,137; 4,915,487; and 4,575,722. Likewise, helmet or head-mounted displays have also been used for motorcycle riders. U.S. Pat. No. 4,988,976 discloses a motorcycle helmet that displays data or information such as speed, time, rpm's, fuel, oil, etc. on the transparent visor (i.e. vacuum fluorescent display) of the rider. Head-mounted displays that are worn in front of the user's eyes or worn as eye spectacles also exist. For example, see the following U.S. Pat. Nos. 5,129,716; 5,151,722; 5,003,300; 5,162,828; 5,331,333; 5,281,957; 5,334,991; 5,450,596 and 5,392,158.
The field of virtual reality also has driven advances in the use of various types of head-mounted displays. For example, see the following U.S. Pat. Nos. 4,636,866; 5,321,416; 5,347,400; 5,348,477; 5,406,415; 5,414,544; 5,416,876; 5,436,765; 5,479,224; 5,473,365; D363,279; 5,485,172; 5,483,307; 5,130,794. See also the publication How Virtual Reality Works, by J. Eddings (Ziff-Davis Press, Emeryville, Calif., 1994), and the site maintained by Rolland (referenced above) relating to telepresence systems and augmented reality.
Advances have also been made in the area of heads-up displays or screens that are not attached to or worn by the user. Most commonly, such systems are employed in automotive or military environments, to provide vehicle performance, weapon status, and other data for the driver or pilot. For examples of such systems, see the following U.S. Pat. Nos. 5,278,696; 4,652,870; 4,711,512; 4,729,634; 4,799,765; 4,927,234; 4,973,139; 4,988,976; 4,740,780; 4,787,711; 4,740,780; 4,831,366; 5,005,009; 5,037,182; 5,231,379; 4,824,228; 4,763,990; 4,669,810; 4,688,879; 4,818,048; 4,930,847; 4,932,731; 5,198,895; 5,210,624; 5,214,413; 5,302,964; 4,725,125; 4,188,090; 5,028,119 and 4,769,633.
Numerous advances have occurred in the specific forms of, and materials used in, heads-up display systems. See, for example, U.S. Pat. Nos. 4,987,410 and 4,961,625 (use of Liquid Crystal Displays (LCDs)); U.S. Pat. Nos. 5,108,479 and 5,066,525 (laminating glass plates or panels); and U.S. Pat. No. 5,457,356 (making a flat panel head-mounted display).
Also pertinent to this invention is the field of eye-tracking to control various computer or imaging functions. Various systems unrelated to the medical field have used eye-tracking for controlling a field of view. For example, see U.S. Pat. No. 4,028,725, which discloses an eye and head tracking system that controls a beam-splitter and retains the high-resolution part of the image in the field of view. The eye-tracking is carried out by infrared detection (i.e. see U.S. Pat. No. 3,724,932). See also U.S. Pat. Nos. 5,287,437; 4,439,755; 4,349,815; 4,437,113; 4,028,725 (referenced earlier) and the article “Optical Design and Analysis of a Head-Mounted Display with a High-Resolution Insert,” referenced above, particularly at footnote 19, which refers to the Eye-tracking Systems Handbook, Applied Science Laboratories, Waltham, Mass. (1992).
Finally, video recording systems for recording scenery and heads up displays have been taught by the prior art. U.S. Pat. No. 5,241,391 to Dodds (“Dodds”) discloses a video camera system that records scene conditions and heads-up displays.
Notwithstanding the large number of articles and patents issued in the area of heads-up or head-mounted displays, there has been no such display that is designed for the special needs of individuals performing detailed but critical tasks on relatively stationary subjects. Such a system would be extremely useful to personnel working in the fields of medicine, forensics, and micro-electronics.
Presently, there is a need for a selectively operable, head-mounted, see-through viewing display system for presenting desired information and/or images to a user, while at the same time allowing the user to view the real-world environment in which he or she operates. There is a further need to provide a convenient selectable viewing system that can be easily controlled by an eye-tracking cursor and speech recognition to control different images or displays on a video monitor or to control a field of view, while keeping the user's hands free to conduct precision operations.